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ABSTRACT 


Technology Issues related to the use of robots as man-extension or telerobot systems in 
space are discussed and exemplified. General considerations are presented on control and 
information problems in space teleoperation and on the characteristics of Earth orbital tele- 
operation. The JPL R&D work in the area of man-machine Interface devices and techniques for 
sensing and computer-based control is briefly summarized. The thrust of this R&D effort is 
to render space teleoperation efficient and safe through the use of devices and techniques 
which will permit integrated and task-level (V intelligent' 1 ) two-way control communication 
between human operator and telerobot machine in Earth orbit. Specific control and informa- 
tion display devices and techniques are discussed and exemplified with development results 
obtained at JPL in recent years. 


1 . INTRODUCTION 


Current practice in robotics divides into two main areas: industrial robotics and robotic teleoperation. 

Industrial robots are used as an integral part of manufacturing processes and within the frame of production engi- 
neering techniques to perform repetitive work in a structured factory environment. The characteristic control 
of industrial robots is a programmable sequence controller, typically a mini- or microcomputer that functions 
autonomously with only occasional human intervention, either to reprogram or retool for a new task or to correct 
for an interruption in the work flow. Teleoperator robots, on the other hand, serve to extend, through mechanical, 
sensing and computational techniques, the human manipulative, perceptive and cognitive abilities into an environ- 
ment that is either hostile to or remote from the human operator. Teleoperator robots or, in today's nomenclature, 
"telerobots" typically perform non-repetitive or singular, servicing, maintenance, repair or rescue work under 
a variety of environmental conditions ranging from structured to unstructured conditions. Telerobot control is 
characterized by a direct involvement of the human operator in the control since, by definition of task require- 
ments, teleoperator systems extend or augment human manipulative, perceptual and cognitive skill which is far 
beyond what is obtainable with today's industrial robots. As a consequence, the human operator interface to a 
teleoperator or telerobot becomes a critical issue. 

Continuous human operator control in teleoperation has both advantages and disadvantages. The main advantage 
is that overall task control can rely cn human perception, judgement, decision, dexterity and training. The main 
disadvantage is that the human operator must cope with a sense of remoteness, be alert to and integrate many infor 
mat ion and control variables, and coordinate the control of one or two mechanical arms each having many (typically 
six) degrees of freedom - and doing all these with limited human resources. Furthermore, in many cases like space 
and deep sea applications, communication time delay interferes with continuous human operator control. 

Modern development trends in teleoperator control technology are aimed at amplifying the advantages and alle- 
viating the disadvantages of the human element in teleoperator control by the development and use of advanced sens 
ing and graphics displays, intelligent computer controls, and new computer-based man-machine interface devices 
and techniques in the information and control channels. The use of model and sensor data driven automation in tele 
operation offers significant new possibilities to enhance overall task performance by providing efficient means 
for task-level controls and displays. 

The purpose of this paper is to discuss and exemplify the technical issues involved in employing robots as 
man-extension or teleoperator systems in space. The primary space applications considered here ;ire u.--- .lanical 
operations in Earth orbit. These include the deployment, servicing, maintenance or retrieval of satellites, the 
handling and assembly of structural elements for creating space station or other large space structures, and the 
maintenance or repair of finished and operational space systems. 

General considerations are presented in Section 2 related to: (i) characteristics of earth orbital environ- 
ment from the viewpoint of remote robot control, (li) control and information problems in remote operations of 
robots with emphasis on human factors involved in the information-control loop, and (iii) data driven automation. 
Specific control techniques are discussed in Section 3 suited to make efficient use of human command and control 
capabilities in task-level controls. Techniques and examples are presented in Section 4 aimed at integrated and 
task-level displays of multidimensional sensory information to aid control decisions. 
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2. GENERAL CONSIDERATIONS 


The use o£ robots as man-extension or teleoperator systems in Earth orbit requires two major considerations. 

The first is related to the specifics of structures and environment In Earth orbit, the second is related to the 
generic nature of operating robots as multi-degree-of- freedom mechanical systems in performing dexterous tasks. 

2.1 Teleoperation in Earth Orbit 

A number of specific conditions must be considered for teleoperation in Earth orbit. Fir ®t, the objects to 
be handled by robot arms are typically large or extended objects. The manipulation of large objects by robot arms 
typically requires the specification and observation of widely separated contact points between object and e, ™iron- 
ment. Second, the structural elements in Earth orbit are typically composed of light materials having low specific 
mass. But the ratio of the robot arm's inertia versus the manipulated object's inertia can vary by orders of mag- 
nitude. Third, the weightless environment in Earth orbit removes the directional effect of gravity. In zero grav- 
ity, dynamically defined "up" and "down" do not exist; things do not "drop down ; contact between 0 ^ e ^ t8 ® nd 
ronment must be established by the robot arm's controller actively. Fourth, the visual conditions in Earth o 
short "day and night" periods, non-diffuse light, highly disparate backgrounds for viewing work scenes, etc. -- 
impose a number of operational constraints. Fifth, dependent upon the physical distance between control station 
and robot in Earth orbit, various communication bandwidth or communication time delay constraints may exist w c 
have an effect upon control and information system design and performance. 

2.2 Information and Control Complexity 

Task-level control of robot arms requires the coordinated motion or force control of several (typically six) 
robot arm Joints while observing a variety of kinematic, dynamic and environmental constraints. Then, to comp y 
with the specifics of a given task, different sensor signals must be interpreted in real time. Furthermore, manip- 
ulation tasks can often be performed in different ways. Hence, robot arm task-level control implies a multilevel 
decision and monitoring process at both the control input and information feedback channels. 

It is known that the human operator's input and output channel capacities are not only limited but also asym- 
metric: the human has much more information receiving (input) channels than information conveying (output channels. 
In this sense, the human operator represents a limiting factor in the complex information and control environment 
of a remotely operated robot. Following this recognition, the general objectives of control, information and man- 
machine interface development for space robots as man-extension systems are: Provide devices and techniques which 

enable the human operator to convey control commands to and receive control feedback from the remotely operate 
robot in comprehensive, integrated and task-level terms and formats. 

2 . 3 Data Driven Automation 

Data driven automation here refers to the use of models and sensing sources through computers in the control 
of remotely operated robots. Data derived from models typically provide a priori information about robot machines 
and tasks. Data derived from sensing sources typically provide on-line Information about robot task performance. 
Data driven automation is inherently flexible since it is programmable. It contrasts the mechanically fixtured, 
rigid or fixed automation. 

Application of robots in space as man-extension systems requires flexibility in both control and information 
management in order to cope efficiently with varying and unpredictable task conditions. The use of data driven 
automation offers significant new possibilities to enhance overall task performance by providing programmable de- 
vices and techniques for task-level controls and displays. 

3. CONTROLS 


Computer controls based on robot arm and task models and on information from sensors integrated with the 
robot arm's end effector permit the development of new devices and techniques which enable the operator to exer- 
cise control commands in comprehensive task-level terms. Computers also allow the use of voice as a new commu- 
nication channel in controlling elements of remote robot systems. 

3. l Interactive Sensor Referenced Manual-Automatic Control 

interactive control signifies here a hybrid control capability which allows that some motions of the remote 
robot arm in work space coordinates are under manual control while the remaining motions in the same work space 
reference frame are under automatic computer control based on sensor information originating from the robot en 
effector. It is noted that, in this hybrid control system, the manual control is in task-level terms, using re- 
solved rate controls, which also require a computer in the control system. The sensor-referenced automatic con- 
trols are also in task-level terms defined within a pre-programmed control menu. 

A pilot computer control system has been implemented at JPL for a six-degree-of freedom robot arm the end 
effector of which is equipped with proximity and force-torque sensors. The sensors ;ud the implementation details 
of this pilot control system are described in Refs. 1 and 2. The general system configuration together with the 
manual and computer control panels are shown in Fig. 1. The manual control Is normally in resolved rate or re- 
solved position mode, using the appropriate computer control algorithms to interpret two three-degree-o - ree om 
joystick inputs. A preprogrammed sensor-referenced automatic control menu is available to the operator who tie 
cides on-line when and which automatic control function should be activated or deactivated. Each automatic con- 
trol function selection can be accomplished by turning a simple on-off switch addressed directly to the control 
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The main purpose of this pilot development project was to study and evaluate- the control hardware and soft- 
ware performance implications of Interactive manual and automatic control in teleoperation. Some of the main 
conclusions are: (t) logic decision nets are the dominating elements in this type of control, and their imple- 

mentation requires special care; (il) the capability of executing both manual and automatic computer controls 
within the same task, function and action formulation frames facilitates the operational integration of human 
and machine logic in teleoperator control. 


3 . 2 Generalized Bilateral Manual Control 

In bilateral, force-reflecting manual control the operator feels the forces and torques acting at the reaot 
robot hand while he manually controls the motion of the robot hand through a master Input device which is called 
the -'.'master arm". In the existing industry practice, the master arm Is a one-to-one size replica of the remote 
slave arm, and. each slave arm must have its own master arm. In most cases the master arm is aech nicaSly couple 
to the slave arm. This is the standard practice In the nuclear Industry. Only in a few cases Is the coupl ing 
implemented electromechanlcally through bilateral servo control. 


A limiting factor for broadening the application of bilateral force-reflecting robot control technology is 
the nature of the master arm. To overcome this limitation, a new form of bilateral, force-reflecting asanual cor 
trol has been implemented at JPL recently. It utilizes a general purpose force-reflecting hand controller 
(Ref. 3). The hand controller is a slx-degree-of-freedom control input device that can be back-driven b? forces 
and torques sensed at the base of the end effector of a remote robot arm. This hand controller is general purpe 
in the sense that it does not have any geometric and dynamic similarity to the slave arm it controls; it Is rx>t_ 
a replica of any slave arm, but it can be coupled to and used for the control of any remote slave arm. 


The positional control relation between the general purpose hand controller and a remote robot arat is estafc 
1 ished through mathematical transformation of joint variables measured at both the hand controller and robot arn 
Likewise, the forces and torques sensed at the base of the remote robot hand are resolved into appropriate hand 

controller motor drive commands through mathematical transformations to give to the operator's hand the same 

"feeling" that is "felt" by the remote robot hand (Ref. A). The complex bilateral mathematical transformations 
are performed by a dedicated minicomputer in real time. These transformations also effect motion synchron lzat Ic 
between hand controller and slave arm, referenced to the slave hand, by hackdriving the hand controller. Overal 
system Implementation is shown In Fig. 2. A preliminary control system analysis and synthesis of this system 

can be found in Ref. 5. Some experimental results are presented in Ref. 6. 


The new form of bilateral manual control of remote robot arms described here generalizes the force-ref lect i 
manipulator control technique. This type of control provides a kinesthetic coupling between operator and reaoce 
robot arm, and can be considered as a combination of "body language" and "reflexive feedback with same basic 
communication primitives as indicated In Fig. 3. Note that these primitives allow task- level^ control . Through 
kinesthetic coupling the operator can command with "feel" and control with a "sense of touch. 


Laboratory experiments are currently conducted at .JPL to determine the effect of weightlessness or the ruma 
arm on this control mode when the operator Is located in Earth orbit. Some preliminary results of thcs« experi- 
ments are described in Ref. 7. 



Fig. 2. Overall system implementation of generalized bilateral manual control. 
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Fig. 3. Integration of graphics, kinesthetic, computer and voice 
man-machine Interfaces. 


3.3 Voice Control 

The use of voice commands to control machines offers a new communication channel which is open and within 
reach most of the time and does not require manual or some specific visual contact between operator and machine 


Advancements in computer-based discrete word voice recognition systems make the direct use of human speech 
feasible for control applications in a teleoperator control station. Several such applications have been deve 
oped at JPI (Ref. 8). The latest application system was developed for the control of the Space Shuttle TV cameras 
and monitors while the operator manually controls the Shuttle robot arm. In this application the operators coul- 
"push" control switches by voice Instead of using fingers. Some Shuttle robot arm tasks are visually very deman,- 
Ing, and can require 50 to 70 commands to four TV cameras and two TV monitors within 15-20 minutes time : rame 
to assure sufficient visual feedback to the operator. The ground control tests at the Johnson Space Center 
(Ref. 9) have shown 96 to 100% voice recognition accuracy for the best test runs and resulted in the foliowing 
major conclusions: (1) the application concept is realistic and acceptable; (li) the use of voice commands in- 

deed contributes to a better man-machine Interface integration; (iii) individual human acoustic characteristics 
and training have a major Impact on system performance. 


Several alternative combinations of control vocabulary words with and without syntax restrictions were devel- 
oped and tested. Altogether thirty-six control switches had to be activated by voice commands. The training 
experiments have shown that the operators prefer simple vocabularies with minimum or no syntactic restrictions. 

To cope with this desire, vocabularies were constructed using concatenated words for full action commands. The 
most successful vocabulary is shown In Fig. 4. It has no syntax, uses concatenated command words, and contains 
only two simple words ("stop" or "reverse") which logically must follow some action commands. Compare t.-e simp.e 
vocabulary to the vocabulary with syntax that also Is shown in Fig. 4. As it turned out, the operators remembered 
and used with higher confidence buzz-word- 1 ike voice commands than words which were embedded into syntactic 
procedures . 


4. DISPLAYS 



pcs It ion/or ientat ion control of the end effector. 


The nonvisual sensor information supplements the visual information and is related to the control of the 
mechanical hand’s contact or near-contact with objects and environment. The nonvisual sensor information provide, 
a combination of geometric and dynamic reference data for the control of terminal position/orientation anc dyn**n~ 
accommodat ion/ccmpliance of the mechanical hand. Non-visual Information related to robot arm control can be ob- 
tained from proximity, touch, slip, and force-torque sensors integrated with the robot arm/hand as illustrated 
in the lower right part of Fig. 3. More on these sensors and their applications can be found m net. 
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Fig. 4. Voice command vocabulary for TV camera/monitor control, using words with syntax 
and concatenated words without syntax. 


Graphics displays of proximity, touch, slip, and force-torque sensor information transform non-visible or 
ha rdl v- v is ib l e events into visually perceivable forms on a graphic terminal. Graphics dispiavs of sensor infor- 
mation can be used in both manual and computer control modes, in a manual control mode the displays are elements 
in the continuum of a real-time control loop in the sense that they guide the operator's continuous control input 
by providing continuous information feedback on the appropriate "external error state" of the robot hand. In 
a computer control mode, the displays represent discrete elements outside the real-time control loop. They pro- 
vide information to the operator prior to the selection and initialization of an appropriate sensor-referenced 
computer control algorithm, and inform the operator about the performance of the control algorithm selected for 
the task at hand. 

The stream of data generated by sensors on a "smart hand” (proximity, touch and force-torque sensors) pro- 
vides multidimensional information, and requires quick (sometimes split-second) control response. In general, 
the control decision required to respond to the data is also multidimensional. This represents a demanding task 
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and heavy workload for the human operator. It is also recognized that the use of information from sensors on 
a "smart hand" often requires coordination with visual information. 

4 . 1 Event Driven Displays 

Event-driven displays can considerably sharpen the Information content of multidimensional sensor data and 
thereby aid the operator’s perceptive and decision making task. 

By definition, event-driven displays map a control goal or a set of subgoals into a multi-dimensional data 
space based on the fact that control goals or subgoals always can be expressed as a fixed combination ot multi- 
dimensional sensory data. Event-driven displays can be implemented by real-time computer algorithms which U) 
coordinate and evaluate the sensory data In terms of predefined events and (ll) drive the graphics display. Flo 
ible display drive algorithms require a variable set of task oriented parameters specifiable by the operator in 
order to match the specific needs of a given control task. (Ref. 10) 

An event driven "smart" display system, developed at JPL, is shown in Fig. 5 together with the proximity 
sensor system which generates the sensory data to he displayed. The figure also shows the measurement definitions. 
The purpose of the "smart" display Is to show the operator the values of range, pitch and yaw errors referenced 
to end effector axes, and also to Indicate whether the combination of these three errors will allow a successful 
grasp of the target. (Ref. 11) 


SQUARE MATRIX CONFIGURATION OF FOUR-SENSOR OPERATION CONCEPT FOR SIMULTANEOUS 

PROXIMITY SENSORS ON FOUR-CLAW B40 EFFECTOR MEASUREMENT OF DEPTH, PITCH AND YAW ERRORS 
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Fig. 5. Proximity sensor system with "smart" e 
robot arm application. 
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ivent driven display for space shuttle 



The graphic display has been built from 10-element linear LEO displays encapsulated In one chip, with indi- 
vidual addressable anode and cathode for each element in the chip. The graphic display resolution is 0.2 inches 
(0.508 cm) per display element in depth, and 1 degree per display element in pitch and yaw errors. The quantita- 
tive value of each error for each bar is increasing away from the center green lamp. Hence, zero error for each 
bar is at the center of the display. This focuses the operator's attention to a single "goal point" on the dis- 
play towards which all error bars should be decreased and where the event indicator "green light" should be on 
for successful grasp. 

Note that depth error is indicated with two identical bars converging in a parallax-type view arrangement 
towards the center green lamp. This renders the display more symmetric and facilitates the distinction between 
angular and depth-error bars. The green light "on" condition Indicates that the existing combination of depth, 
pitch and yaw errors will allow a successful grasp. 

The graphic display also contains a tone generator. It provides a "warning tone” (a short beep tone) when 
the target reaches the sensing range or leaves the sensing range. The maximum depth sensing range shown on the 
dtaplay is 6 Inches (or 15 cm). Pitch and yaw errors are indicated in the range of +15 deg. 

The numeric display resolution is 0.1 inches (0.254 cm) in depth error and 0.5 deg in angular errors. It 
also has the "green success lamp". 

The "smart" event-driven displays are controlled by a single board Intel 80/20 microprocessor which linear- 
izes the sensor data and processes the linearized data through a preselected "success algorithm". An appropriate 
"success algorithm" can be selected in the computer through a switch. The algorithms can be referenced to alter- 
native roll orientations of the end effector pitch and yaw axes, and car utilize alternative numeric definitions 
for "successful grapple envelope" in terms of maximum and minimum values of allowable depth, pitch and yaw errors. 

4 . 2 Event Controlled Displays 

Event-controlled displays extend the capabilities of event-driven displays by automat ical ly effecting changes 
between data displays and data formats on a graphics monitor. 

The need for different types of sensor data displays or for different formats of data displays typically 
arises in a logical sequence in remote robot control tasks. For example, when proximity sensor data are needed 
then normally there is no need for touch or force-torque sensor data, or vice versa. This sequential logic in 
the need of sensor information can be utilized to switch automatically between different data displays or formats. 
Following this concept, event-controlled displays have been implemented at JPL (Ref. 12). In the Implemented 
examples predefined changes in sensor data automatically effect changes In display modes, formats and parameters, 
matching the need for a particular information to different phases of the task. Event-controlled displays require 
tne Implementation of state transition nets In real-time computer programs based on event detection logic. 

Event controlled or automatic display mode/format switching can alleviate much of the display control work- 
load for the operator. 


5. CONCLUSIONS 


Automation In teleoperation is distinguished from other forms of automated systems by the explicit and active 
inclusion of the human operator In system control and Information management. Such active participation by the 
human, interacting with automated system elements in teleoperation, is characterized by several levels of control 
and common 1 cat Ion, and can be conceptualized under the notion of "supervisory control" as discussed in Ref. 13. 

The man-machine interaction levels in teleoperation can be considered in a hierarchic arrangement as outlined 
In Ref. 14: (l) planning or high level algorithmic functions', (11) motor or actuator control functions, and (ill) 
environmental interaction sensing functions. These functions take place in a task context in which the level 
of system automation is determined by (a) the mechanical and sensing capabilities of the telerobot system, (b) 
real time constraints on computational capabilities to deal with control, commu n icat icq and sensing, (c) the 
amount, format, content and mode of operator interaction with the telerobot system, (d) environmental constraints, 
like task complexity and (e) overall system constraints, like operator's skill or maturity of machine intelligence 
t echn iques . 

Some advances have been made in teleoperator technology through the introduction of various sensors, com- 
puters, automation and new man-machine interface devices and techniques for remote manipulator control. The de- 
velopment of dexterous mechanisms, smart sensors, flexible computer controls, intelligent man-machine interfaces, 
and Innovative system designs for advanced teleoperation is, however, far from complete, and poses manv Interdis- 
ciplinary challenges (Refs. 15-17). It should also be recognized that the normal manual dexterity u humans is 
mo'e a "body" skill than an intellectual one. The man-machine interface philosophy embodied in the torce- 
reftecting master-slave manipulator control technology has been founded mainly on this fact. Advanced teleopera- 
tion employing sensor-referenced and computer-controlled manipulators shifts the operator-te l erohot interface 
from the body (analog) level to a more intellectual language- 1 ike (symbolic) level. Research efforts tor develop- 
ing new man-machine interface technology for advanced leleoperat ion will have to render the language-like symbolic 
interface between human operator and telerobot as efficient as the conventional analog interface. This remark 
also applies to operator Interface development for procedure execution aids and for expert systems in teleoperator 
action planning and error recovery. 
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